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T he Child Welfare League of 
America’s (CWLA) National 
Advisory Committee on 
Residential Services recommended that 
CWLA develop a set of uniform defini- 
tions related to restraint and seclusion 
and a list of primary and secondary data 
points to be used in data collection. 

The Committee viewed the lack of 
uniformity in definitions and data ele- 
ments as barriers to data collection, 
information sharing, benchmarking, and 
advancing efforts to reduce and eventual- 
ly eliminate restraint and seclusion. By 
instituting consistent definitions and data 
points, children’s residential facilities, 
jurisdictions, and states will be able to 
access and analyze the data and compare 
their use of restraint and seclusion to other 
similar facilities locally and nationally. 

In November 2003, CWLA held a 
National Definitions and Data Collec- 
tion for Residential Facilities meeting in 
Miami, Florida. Attendees included 
CWLA members, providers from around 
the country, national accreditation organ- 
izations, behavior support and interven- 
tion training organizations, and other 
individuals and national organizations 
concerned with the use of restraint and 
seclusion. The result was a set of defini- 
tions concerning restraint and seclusion 



and a list of primary and secondary 
elements for data collection. 

Following the meeting, CWLA’s 
National Advisory Committee on 
Residential Services, and attendees from 
the National Definitions and Data 
Collection for Residential Facilities, 
offered feedback, and the document was 
revised based on their comments. 

The complete set of definitions 
and primary and secondary data points 
can be found on CWLA’s Residential 
Group Care website at www.cwla.org/ 
programs/groupcare. Providers, jurisdic- 
tions and states deciding to incorporate 
these definitions and data points into their 
policies, procedures, and practices may 
create their own network for information 
sharing, and benchmarking. In addition, 
these efforts could help the field reduce 
and eventually eliminate restraint and 
seclusion. 

The League will continue to advocate 
and seek creative and diverse funding 
sources to establish a National Resource 
Center to collect and disseminate data 
and technical assistance for children’s 
nonmedical residential facilities. 

Lloyd Bullard is Project Director, Best 
Practices in Behavior Support and 
Intervention, CWLA. 
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CWLA Publishes Best Practices in 
Behavior Support and Intervention 
Assessment Instrument 



By Nupur Gupta 

C WLA’s new Best Practices 
in Behavior Support and 
Intervention Assessment was 
developed to help agencies strengthen 
their behavior support and intervention 
policies, procedures, and practices by con- 
ducting a thorough self-assessment. With 
the increasing awareness of risks posed by 
restraint and seclusion, agencies should 
consider high rates of restraint and seclu- 
sion evidence of treatment failure and 
thus strive to improve their policies and 
procedures so they can reduce the use of 
emergency physical interventions. 

The Substance Abuse Mental Health 
Services Administration hopes to reduce 
and eventually eliminate the use of restraint 
and seclusion among institutional and 
community-based residential settings, and 
it has funded this project with that goal 
in mind. 

Strong leadership is essential to elimi- 
nating the unnecessary use of restraint and 
seclusion, and self-assessment is the first 
step leaders must take. The assessment 
instrument covers five major areas agencies 
should review when assessing their behav- 
ior support and intervention policies and 
practices: ethical and legal framework, 
administration and leadership, continuum 
of intervention, medical issues, and profes- 
sional development and support. 

When conducting the assessment, 
agencies must keep in mind that careful 
evaluation includes closely reviewing 
agency documentation, as well as seeking 
information from a group of staff mem- 
bers, children, and stakeholders from 
each unit being evaluated. 

Each section of the assessment provides 
several standards, along with supporting 
indicators, all of which must be carefully 
reviewed to accurately rate the agency’s 
implementation of those policies. If an 
agency meets all of the indicators listed, 
the standard is consistently reflected, 
which implies all the practice standards 



are consistently followed and only very 
minor or no improvements are needed. 

If an agency misses only one or two 
of the practice indicators, that standard is 
partially reflected and requires improve- 
ment. If an agency misses three or more 
of the practice indicators, that standard is 
poorly reflected and requires significant 
improvement. If an agency meets none of 
the indicators, written policies and proce- 
dures for that standard are absent and 
clearly inadequate. 

For example, under Ethical and Legal 
Framework, item number three states: The 
agency has a written statement outlining 
families’ rights. Seven indicating factors 
follow, which include: 

• Families have the right to receive writ- 
ten information, which the agency will 
translate if necessary, about the 
provider’s policies and procedures. 

• Families have the right to be involved 
in the assessment and service planning. 

• Families have the right to be notified 
following any use of seclusion or 
restraint, suicide attempt, medical 
emergencies, or other seminal event. 
The back of the assessment instrument 

contains an answer sheet to track ratings 
for all questions. The last part of the 
instrument includes an appendix detailing 
reference page numbers in the CWLA 
Best Practice Guidelines for Behavior 
Management corresponding to each 
standard and its supporting indicators. 

With almost 50 questions to guide 
agencies in assessing the various policies 
and procedures an agency may have or 
need to revise, the assessment tool aims 
to provide agencies with the potential and 
right direction to reduce, and eventually 
eliminate, the use of restraint and seclu- 
sion. To learn more about the instru- 
ment or order a copy, visit ww.cwla.org/ 
programs/behavior/pubs. htm. 

Nupur Gupta is a Program Assistant at CWLA. 
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Initial Findings of an Evaluation 
of a Trauma Recovery Framework 
in Residential Treatment 



By Jeanne C. Rivard 



T his article briefly describes the 
initial findings of an evaluation 
of the Sanctuary Model (Bloom, 
1997, 2003), an intervention designed to 
address the special treatment needs of 
youth with emotional and behavioral 
disturbances and histories of maltreat- 
ment or exposure to domestic and com- 
munity violence. 

The Sanctuary Model integrates 
trauma theories (Bloom, 1997), an 
enhanced therapeutic community philos- 
ophy (Bloom, 1997), and recommended 
child treatment strategies that address 
post-traumatic symptoms, developmen- 
tal disruptions, and unhealthy accom- 
modations to traumatic experiences 
(Friedrich, 1996). 

A fundamental premise of the inter- 
vention is that the treatment environ- 
ment is a core modality for modeling 
healthy relationships among interde- 
pendent community members. This 
trauma-informed systems approach 
was described in a previous issue of 
Residential Group Care Quarterly 
(Bloom, 2003). In the setting of a 
residential treatment center, the model 
was operationalized through: 

• a series of staff dialogues and self 
evaluations of residential units’ 
structure and functioning, 

• staff training and ongoing technical 
assistance, 

• twice-daily community meetings co- 
led by staff and youth, 

• a range of psychoeducation exercises 
that staff use in their daily interac- 
tions with youth, and 
• weekly psychoeducation groups 



(Duffy, McCorkle, &; Ryan, 2002) 
to teach the knowledge and skills 
needed to progress through four 
stages of recovery (i.e., safety, emo- 
tions management, loss and grieving, 
and future orientation; Foderaro & 
Ryan, 2000). 

The evaluation project was conduct- 
ed as a partnership between researchers 
of Columbia University School of Social 
Work, the Center for Trauma Program 
Innovation of the Jewish Board of 
Family and Children’s Services in New 
York City, and the model developer, 
Sandra Bloom. 

The project originated from the host 
agency’s desire to enhance its service 
delivery by incorporating a new approach 
to meeting the special needs of youth in 
its residential treatment programs. The 
research component was funded through 
an exploratory/developmental research 
grant by the National Institute of 
Mental Fdealth as part of an initiative 
to promote research on interventions 
for youth violence. 

Evaluation Methods 

The project took place in a suburban 
community outside New York City 
where the Sanctuary Model is being 
implemented in three residential treat- 
ment programs on one large campus. 

The model was piloted in four residen- 
tial units that self-selected to participate 
in the initial phase of the project. 

The staff training protocol and man- 
ual was developed and piloted between 
February and August 2001, then four 
additional residential treatment units 



were randomly assigned to implement 
the Sanctuary Model the following fall. 
Eight other units that provided the stan- 
dard residential treatment program 
served as the control group. 

Changes in the therapeutic commu- 
nities and in youth were assessed every 
three to six months through April 2003. 
Results of the Sanctuary Model units 
were compared to results of units with 
standard residential services. 

Although the Sanctuary Model was 
in a very early stage of implementation, 
the evaluation was guided by hypotheses 
that projected what specific outcomes 
were expected to occur in the therapeutic 
communities and in youths. We expected 
to find greater changes over time in the 
Sanctuary Model units than in the stan- 
dard residential treatment units in the 
following areas: 

Therapeutic Communities 

• Increase in perceived sense of 
community/cohesiveness 

• Increase in democratic decision- 
making and shared responsibility 
in problem-solving 

• Reduction in critical incidents and 
use of physical restraints 

Youth 

• Reduction in traumatic stress 
symptoms 

• Increase in level of self-esteem 

• Greater internal locus of control 

• Greater use of social network 

• Improvement in decisionmaking and 
problem-solving skills 

• Decrease in aggressive behavior 
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Youth Demographics and History 

Demographic and historical data were 
obtained from clinical records at base- 
line. History of abuse and neglect was 
abstracted from these records using the 
Maltreatment Classification System 
developed by Barnett, Manly, and 
Cicchetti (1993). Exposure to violence 
in home, community, or neighborhood 
was assessed through the My Exposure 
to Violence instrument (Buka, Selner- 
O’Hagan, Kindlon, & Earls, 1997). 

Youth (N = 165) ranged in age 
from 12 to 20 years, with a mean age 
of 15. Seventy-three percent were male 
and 27% were female. Fifty-one per- 
cent were black, not Hispanic; 34% 
were Hispanic; 11% were white; and 
4% were Asian, Pacific Islander, or 
biracial. 

Youth averaged six prior place- 
ments, including an average of three 
psychiatric hospitalizations. Thirty-four 
percent had experienced at least one 
substantiated incident of physical 
abuse, 12% had at least one substanti- 
ated incident of sexual abuse, and 45% 
had at least one substantiated incident 
of neglect. Most youth experienced 
multiple incidents of maltreatment. 

A self-report of lifetime exposure 
to violence showed that 42% of youth 
had seen someone else attacked with 
a weapon, and 23% had been attacked 
with a weapon themselves. Twenty 
percent reported having seen someone 
else shot, and 11% reported having 
been shot at (Rivard et ah, 2003). 

Implementation of the 
Sanctuary Model 

Our evaluation emphasized an assess- 
ment of the processes of model imple- 
mentation. Could we successfully oper- 
ationalize and measure incremental 
changes in the therapeutic communi- 
ties? Would staff and youth understand 
the model, accept it, and see its value? 
Would successful implementation actu- 
ally lead to change in staff and youth 
behaviors? 



We documented progress in imple- 
menting the model using consultants’ 
process notes and periodic reviews of 
the Sanctuary Project Implementation 
Milestones checklist, which contained a 
list of observable criteria. Researchers 
gathered qualitative data on staff per- 
ceptions of the course of implementa- 
tion, and challenges in implementing 
the model using focus groups. We also 
used focus groups to see youths’ under- 
standing of the model and their impres- 
sions of its effectiveness. 



Would staff and youth under- 
stand the Sanctuary Model, 
accept It, and see Its value? 



Across the eight units that imple- 
mented the model, scores on the 
Sanctuary Project Implementation 
Milestones criteria ranged from 66% 
to 92%, with a mean of 78%. The 
slowest and most difficult component 
implemented was the weekly psycho- 
education group. We saw greater 
implementation among those units 
exposed to the model longer, those 
serving girls, and those with leaders 
who had greater enthusiasm and 
commitment to the model. 

Through focus groups with staff, 
we learned one of the most important 
findings concerned factors that facili- 
tated not only implementation, but 
also consistency in using the model. 
These factors included building in 
structured times for discussing imple- 
mentation and team-building, procedu- 
ralizing use of the psychoeducation 
tools, general openness of staff and 
youth to the model, acknowledging 
small successes to build enthusiasm, 
helping youth gain a deeper under- 
standing of the trauma recovery frame- 
work, group cohesion, providing com- 
munity-level incentives for positive 
community behaviors, and program 



leadership (Rivard et ah, in press). 

From youth focus groups, we 
learned that they both understood and 
thought they could benefit more from 
the Sanctuary Model in units where 
there was stronger implementation. 

Therapeutic Community 
Outcomes 

The short form of the Community 
Oriented Programs Environment Scale 
(COPES; Moos, 1996) was used to 
assess the extent to which units were 
operating as therapeutic communities 
along selected dimensions. Trends 
in the frequency of critical incidents 
were then measured by analyzing data 
from the agency’s management infor- 
mation system. 

We found no significant differences 
between the Sanctuary Model units 
and the standard residential treatment 
units during the first two waves of 
measurement. By the final wave of 
measurement, however, we found sig- 
nificant differences between the groups 
via independent t-tests, with the 
Sanctuary Model units improving on 
the following constructs of the COPES: 
support {p < .05), spontaneity (p < .01), 
autonomy (p < .05), personal problem 
orientation {p < .05), safety (p < .05), 
and in the total score (p = .001). We 
are still analyzing data measuring trends 
over time in the frequency of critical 
incidents that occurred in the residen- 
tial treatment units. 

Youth Outcomes 

The following instruments were used 
to assess youth outcomes that were 
hypothesized to be responsive to the 
Sanctuary Model: Child Behavior 
Checklist (Achenbach, 1991), the 
Trauma Symptom Checklist for 
Children (Briere, 1996), the Rosenberg 
Self Esteem Scale (Rosenberg, 1979), 
the Nowicki-Strickland Locus of 
Control Scale (Nowicki & Strickland, 
1973), the peer form of the Inventory 
of Parent and Peer Attachment 




(Armsden & Greenberg, 1987), the 
Youth Coping Index (McCubbin, 
Thompson, & McCubbin, 1996) 
and the Social Problem Solving 
Questionnaire (Sewell, Paikoff, & 
McKay, 1996). 

Although baseline data were col- 
lected for 165 youth, substantial attri- 
tion occurred due to youth being dis- 
charged through usual program opera- 
tions. By study end, 87 youth yielded 
three waves of data (i.e., baseline, three 
months, and six months). No signifi- 
cant differences were found comparing 
baseline and three-month measures. On 
repeated analyses (N = 87) comparing 
baseline and six-month outcomes, how- 
ever, we found a few differences by 
time and group, favoring youth in 
the Sanctuary Model units. These were 
on the incendiary communication/ 
tension management construct of the 
Youth Coping Index {p < .05), locus of 
control {p = .15), and the verbal aggres- 
sion construct of the Social Problem 
Solving Questionnaire (p = .15). 

Discussion 

Results were modest and consistent 
with a newly implemented interven- 
tion, especially considering that rates 
of implementation varied across units. 
The few positive youth findings offer 
promise that full implementation may 
yield greater youth benefits. 

The finding that the treatment 
environments of the Sanctuary units 
were functioning at significantly higher 
levels than the standard residential 
units by the final wave of data collec- 
tion suggests that implementation was 
becoming stronger with time. 

In studying children’s service sys- 
tems, Glisson & Hemmelgarn (1998) 
found that organizational climates with 
greater job satisfaction, fairness, coop- 
eration, personalization, and lower lev- 
els of conflict were associated with 
both service quality and positive out- 
comes in children’s psychosocial func- 
tioning. These findings validate the 
current project’s intensive program 



development efforts aimed at strength- 
ening the treatment environment for 
the benefit of staff and youths. More 
analyses of the data will follow. 

Some of the most important les- 
sons learned from this project focus on 
the need to support implementation 
efforts with more intensive onsite tech- 
nical assistance, promote ongoing eval- 
uation to assess change in the treat- 
ment environments and youth over 
time, and incorporate the use of brief 
behavior checklists that can be used as 
part of the regular program operations, 
and that may be more sensitive to 
change than measures of three-month 
self-reports of youth. 

Jeanne C. Rivard, PhD, is Senior Research 
Analyst, National Association of State 
Mental Health Program Directors Research 
Institute. She can be reached at 703/739- 
9333, ext. 146, or jeanne.rlvard@nri-inc.org 
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Are Point and Level Systems 
the Answer? 



By Lloyd B. Bullard 



B ehavior modification techniques, 
such as point and level systems, 
are a fundamental therapeutic 
approach that managers, therapists, and 
child care workers use in treatment and 
care programs for children in residen- 
tial facilities. 

Behavior modification is thought to 
offer proven guidelines and procedures 
for diagnosing, treating, and managing 
children in residential care (Buckholdt & 
Gubrium, 1980). Unfortunately, residen- 
tial treatment programs in the United 
States rely too much on point and level 
systems to bring about behavioral 
change, often sacrificing opportunities to 
build relationships between youth and 
staff members. 

Point and level systems are one of 
the most common behavior modification 
programs used in residential care facili- 
ties (VanderVen, 1999). Point and level 
systems are used to manage behavior 
that is both observable and accessible to 
staff intervention by concentrating on 
overt behavior as opposed to internal 
dynamics (Cohen, 1986). The main 
focus of a point and level system is to 
control every aspect of a child’s actions 
in the milieu, rather than fostering 
development of internal control through 
interpersonal interactions with staff 
(VanderVen, 1999). 

Point and level systems essentially 
award points to children for positive 
behavior and deduct points for 
negative behavior (VanderVen, 1999). 
Staff constantly monitor the behavior of 
residents (Buckholdt & Gubrium, 1980) 
and points are tallied periodically to 
ascertain children’s compliance with the 
current system and determine their sta- 
tus level (VanderVen, 1995). 



Based on points earned, a child may 
be promoted to a higher level; points 
subtracted due to rules infractions may 
cause a child to be demoted to a lower 
level. With each level increase a child 
receives additional privileges, and with 
every demotion the child loses privileges. 
Earning and totaling points may appear 
to be an objective method for determin- 
ing a child’s level. But by including such 
things as the child’s ability to follow 
rules and a willingness to address indi- 
vidual behavior goals, the process 
becomes increasingly subjective. 

Failures are a part of every 
child’s normal development, 
but point and level systems 
may encourage children to 
view failure in a negative light. 



Durrant (1993) points out that 
level systems may appear to be firmly 
established on the notion of rewarding 
success and positive behavior, but may, 
in fact, contribute to failure. Failures are 
a part of every child’s normal develop- 
ment, but point and level systems may 
encourage children to view failure in a 
negative light. 

Many children present problem 
behaviors, and child care workers are 
left with the difficult task of managing 
and helping the child to manage his or 
her own behavior. This usually results in 
the child not earning points nor achiev- 
ing a higher level with greater privileges. 
Many children, therefore, experience an 
even greater sense of failure within point 
and level systems (Durrant, 1993). 



One of the most significant factors 
that should be integrated in the delivery 
of residential services is favorable inter- 
personal interactions between staff and 
residents (Garfat, 1994; Krueger, 1991). 
Interpersonal interactions help establish 
and build relationships between staff 
and residents. Such interactions are 
essential in fostering trust and building 
self-esteem among youth. Once a bond of 
trust is established, self-esteem increases 
and openness to positive development 
persists. Conversely, the absence of inter- 
personal relationships may have an 
adverse effect. 

Since the 1960s, this country has 
experienced a significant increase in new 
approaches and models for managing 
and treating emotionally disturbed chil- 
dren. In spite of this, no consensus 
points to the best approach or model 
(Bertolino & Thompson, 1999). None- 
theless, many of these models use behav- 
ior modification techniques to induce 
the desired behavior. 

Although research examining the 
effectiveness of point and level systems 
is scant, several well-known leaders in 
the field have touched on the subject. 
Some researchers believe point and level 
systems have many limitations and 
are not adequate tools for supplying 
residents with personal interactions 
(Armstrong, 1993; Buckhodt & 
Gubrium, 1980; Durrant, 1993; 

Fox, 1994; Goldfried & Castonguay, 
1993; VandarVen, 1993, 1995, 1999). 

Considering the main objective 
of a point and level system is to control 
a child’s behavior, the presence of sub- 
stantial interpersonal interactions is 
unlikely. Yet interpersonal interactions 
are clearly necessary for forming 
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relationships, a fundamental component 
of child and youth work (Abel & 
Nelson, 1990; Fewster, 1990; Krueger, 
1991; Rose, 1991; VanderVen, 1999). 

Although Mclnnis & Marholin 
(1977) do not specifically mention point 
and level systems, their research indi- 
cates that this kind of individualized 
program is difficult to implement success- 
fully within residential facilities. One 
must ask whether point and level systems 
provide sufficient interpersonal interac- 
tions for residents in these settings. 

In fact, point and level systems can 
provide sufficient interpersonal interac- 
tions for the normal development of 
adult-child relationships. But this would 
require workers to consistently commu- 
nicate with the child and provide guid- 
ance and reinforcement. 

This does not suggest that point and 
level systems are the best approach for 
producing positive outcomes. Agencies 
using various point and level systems 
may have very different results. The 
agency’s leadership, organizational cul- 
ture, staff investment, population served, 
and the methodology used to implement 
the system affect how these systems 
or any others might function within 
an agency. 



Interpersonal Interactions are 
clearly a necessary condition 
for forming relationships, a 
fundamental component of 
child and youth work. 

Behavior models must be individu- 
alized and must take into account the 
developmental stage of each child 
(VanderVen, 1995). Rather than require 
children to conform to the model, the 
model must adapt to the unique needs of 
every child. In addition, the behavior 
model must be as simple as possible; any 
child or youth in the program should be 



able to understand the model and 
explain it. Many point and level 
systems are accompanied by a large 
handbook explaining systems that may 
be difficult to read and hard to under- 
stand; this practice must be revised 
(Gutkind, 1993). 

It’s particularly crucial that point 
and level systems do not give conse- 
quences to children for every rule infrac- 
tion (VanderVen, 1995, 1999). Rather, 
they should rely more on natural conse- 
quences, discussions with the children 
about the reason for their behavior, 
and analysis of the need the child was 
attempting to fulfill. 

To create and develop an appropri- 
ate behavior model for children, agencies 
must provide residents with program- 
ming that offers training in everyday 
skills and social skills, challenging and 
meaningful activities that increase coping 
skills and team cohesiveness, and some 
socially redeeming qualities. In addition, 
staff must be invested in providing these 
children and youth with quality care. 

And that’s not enough. The agency 
must also invest in its employees, provid- 
ing them with appropriate high-quality 
training in communication, relationship 
building, behavior support, active listen- 
ing, team building, and cultural diversity. 

Many agencies aren’t interested in 
using point and level systems for a num- 
ber of reasons, but they’re still faced 
with the challenge of selecting or devel- 
oping a behavior model. A plethora of 
other alternatives are available to those 
agencies not interested in using point 
and level systems. 

Whether an agency is selecting or 
developing a behavior model, the first 
step should include a review of the 
agency’s mission and philosophy, which 
should embrace an overall goal for 
clients served. This process should not 
be limited to administrative staff, but 
should include all disciplines and staffing 
levels. 

Many agencies make the mistake 
of starting their selection process for a 



behavior model by reviewing the numer- 
ous models currently available. By clear- 
ly understanding its mission and philoso- 
phy, as well as the agency’s desired out- 
comes for children and youth, an agency 
can develop a framework to achieve 
those outcomes. 



Is the model transferable to 
the child’s next placement or 
home? Does the model create a 
realistic approach for address- 
ing the child’s behavior? Can 
all staff Implement the model 
consistently? 

The next step requires the agency to 
decide whether to opt for one of the 
available models, such as the teaching- 
family model, guided group interaction, 
positive peer culture, positive youth 
development, risk and resilience, caring 
profile, or numerous others, including a 
design of their own. If the agency choos- 
es one of the available models, it may 
need to alter the model to meet clients’ 
needs. The agency must ensure that the 
model does not require residents to fit 
into it, but rather that the model can be 
adapted to the client’s individual needs 
(CWLA, 2002). The agency should also 
answer the following questions: Is the 
model transferable to the child’s next 
placement or home? Does the model 
create a realistic approach for addres- 
sing the child’s behavior? Can all staff 
implement the model consistently? 

Regardless of the abundance of 
available models, an agency may decide 
to develop and implement its own 
model. No matter which track the 
agency chooses, direct care workers, 
clinical staff, administrative staff, and 
any other staff responsible for the super- 
vision and care of the clients should 
actively participate in the process. 





To achieve staff buy-in, an agency 
must get equal participation from all 
staff members. The success of any 
behavior model depends largely on 
the staff’s investment. Involving the 
direct care workers at the onset is 
extremely important, as they are 
the ones who spend the most time 
with residents and will be primarily 
responsible for implementing the 
model. Lastly, for any behavior model 
to be successful, an agency must to 
incorporate the genuine involvement 
of children. 

Lloyd B. Bullard Is Project Director, 

Best Practices in Behavior Support & 
Intervention, CWLA 
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Best Practices in 
Behavior Support 
and Intervention 
Assessment 

This assessment instru- 
ment will help agencies 
improve their behavior 
support and intervention 
policies, practices, and 
procedures. Agencies that 
perform the self-assess- 
ment will be able to pin- 
point how they can ensure 
the safety of their clients 
and workers without caus- 
ing children more harm 
than they have already 
experienced. See page 
2 of this issue for more information. To purchase the Child Welfare 
League of America’s Best Practices in Behavior Support and Intervention 
Assessment, for $8.95 plus shipping/handling and applicable sales tax, 
there are five easy ways to order: CWLA, P.O. Box 932831, Atlanta, 

GA 31193-2831; Phone; 800/407-6273; Fax: 770/280-4160; or Online: 
www.cwla.org/pubs 
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Point/Counterpoint 



Q « Is Residential Care a 
• Cost-Effective Service? 



T) T'V T' I Most residential care providers are 

J. J.X ^ X • badly underfunded — it’s almost 

impossible to find a residential provider that’s reimbursed 
at its level of cost. In spite of the fact that it’s underval- 
ued and underfunded, however, residential care is a cost- 
effective service that is necessary and effective. 



COUNTERPOINT: 

Residential care is a costly, overused service that does not 
generally produce positive outcomes. Although residential 
care is useful under limited circumstances, the funds spent 
on it would, in general, be better used supporting birth and 
foster families. 



By Bill Powers 



By Richard P. Barth 



M rs. R. told all of us earlier in the day she was 
most definitely not going to cry. Now it’s time 
for the final goodbye, the last in a series of 
final goodbyes, and we’re all standing in the lobby of our 
residential treatment center, crying. Mrs. R. tells us we 
are miracle workers, that she had given up on seeing her 
son reach his 1 8th birthday, and that we have restored 
him to life. She repeats that we just don’t seem to appre- 
ciate what we have done for her family. 

Similar, if less dramatic, scenes play out every day 
across the country. And this scene is at the very heart of 
this discussion: Why do we consistently undervalue a 
treatment service that restores life to “lost” children and 
their families? How does a service that provides hope 
to often voiceless children and their families justify 
its expense? 

To draw comparisons between physical and behav- 
ioral health care, years of successful cardiac, neurological, 
and organ transplant surgeries are no longer questioned 
in spite of their relatively high cost. Yet in behavioral 
health care for children, particularly for those in residen- 
tial treatment centers, we find ourselves constantly 
defending the cost of equally important life-saving servic- 
es. Why? Is it the lack of publicity regarding the positive 
outcomes of residential treatment? Is it the voiceless 
nature of the population served? 

Critics of residential treatment often cite its high 
cost and lack of hard outcome data. Yet given its high 
cost, entry into residential treatment is almost always 
preceded by a long list of failures in less restrictive, less 

see POINT, page 10 



E fforts to contrast the strengths and weaknesses of res- 
idential and foster care are long and storied (Wolins 
& Piliavin, 1964). These deliberations have been 
largely conceptual rather than evidentiary — highlighting the 
potential risks and benefits of each approach. Until recently, 
there wasn’t enough scientific evidence to fully inform this 
debate. 

Even now, large unanswered questions remain about the 
effectiveness of residential care in contrast to alternatives. 
Cost-effectiveness comparisons are the most limited, 
because they require estimates of the size of the improve- 
ment in outcomes for residential care and its alternatives, as 
well as short- and long-term benefits. 

A careful treatment of this question requires bundling 
some rough estimates and overgeneralizations into an argu- 
ment. Still, the resulting conclusion is reasonably robust: 
Residential care consumes a substantial portion of the 
nation’s children’s services budget yet does not offer com- 
mensurate benefit to youth, their families, or society. 

Research comparing the outcomes of residential care 
with other community-based services is beginning to sup- 
port a strong case that residential care is not a comparably 
effective approach to serving youth. Direct experimental 
comparisons between hospitalization and multisystemic 
family therapy (MST) and between MST and incarceration 
(Henggeler, et ah, 2003, Borduin, et ah, 1995) show that 
MST works as well or better. 

The same is even more clearly true in the randomized 
control trials comparing multidimensional treatment foster 
care (MTFC) against group care for juvenile offenders. At 

see COUNTERPOINT, page 11 
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expensive treatment settings. 
Unfortunately, this failure path not 
only rules out less expensive treatment 
options, but also directly contributes to 
the high cost of residential treatment. 
Each successive failure — sometimes as 
many as 7 to 10 — reinforces the child’s 
lack of faith and trust in the system. 

Restoring a lost childhood to these 
youth who have bounced through the 
system requires a large cadre of caring, 
trained professional staff. Reaching 
families, a critical component of success- 
ful residential treatment, requires good 
detective work and enormous patience 
and persistence. 

Furthermore, damage caused by 
years of abuse and neglect, compounded 
by repeated failures in less restrictive 
settings, is not easily or quickly undone. 
Each child and his or her family must 
be met on their own terms to engage 
in a therapeutic relationship. Gains 
achieved through hard work in the resi- 
dential setting must be replicated and 
transferred to the home. Once again, 
this transfer of knowledge and skills 
requires truly caring and experienced 
staff working directly with families. 

To accomplish this complex and 
difficult work, residential treatment 
centers usually receive less than 90% of 
their true costs from public funding 
sources. Fundraising and endowment 
draws, if there is an endowment, must 
make up the difference. Faced with the 
constant threat of economic extinction, 
residential treatment providers priori- 
tize direct services to children and their 
families. As a direct result, research that 
would produce meaningful outcome 
data is rarely funded or is severely 
underfunded. 

In spite of these funding limita- 
tions, residential treatment providers 
are increasingly conducting their own 
outcome studies and collaborating with 
peer agencies to produce more statisti- 
cally significant comparison studies. 

And they’re paying attention to care- 
fully matching demographics to avoid 
drawing conclusions on noncomparable 
populations, a failure of many earlier 
studies. Outcome data across life 
domains are commonly collected at 6, 
12, and 18 months post-discharge. 



A recent CWLA study of the 
research literature reveals that 
characteristics of residential 
treatment centers that have 
been correlated with long- 
term positive outcomes 
include high levels of family 
involvement, supervision and 
support from caring adults, 
a skill-focused curriculum, 
service coordination, develop- 
ment of individualized treat- 
ment plans, positive peer 
influences, enforcement of a 
strict code of discipline, a 
focus on building self-esteem, 
a family-like atmosphere, 
academic support, presence 
of community networks, a 
minimally stressful environ- 
ment, and comprehensive 
discharge planning. (Bullard 
& Johnson, in press). 

In spite of the increasing emphasis 
nationally on collecting meaningful out- 
come data, residential treatment contin- 
ues to be a seriously undervalued treat- 
ment service. Could the disparity in 
public recognition and subsequent ade- 
quate funding also be related to the tar- 



get population? Although the high cost 
of open heart surgery is borne by all 
socioeconomic classes, including corpo- 
rate executives, residential treatment’s 
target population comprises voiceless, 
disenfranchised children and their des- 
perate families. 

Our mission is clearly defined. We 
must continue to gather and publicize 
meaningful outcome studies on the ben- 
efits of residential treatment. We must 
continue to search for common best 
practices that produce these beneficial 
results. We must not allow critics and 
cynics to deter us from providing care 
for Mrs. R. and other parents desper- 
ately seeking help for their children. 

We must continue to support these 
children, our families, and each other. 

Bill Powers is Executive Director, Bonnie 
Brae, Liberty Corner, NJ. 
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two years' follow-up, the MTFC group 
was in the normative range, whereas 
rates for group care youth were at least 
four times higher (Eddy, Whaley, & 
Chamberlain, 2004). Many youth who 
were thought to be best served in group 
care did as well or better in alternative 
forms of care. These findings make 
dubious the idea that a child needs resi- 
dential care. A child may need special- 
ized or intensive services, but these 
services may not be best provided in a 
group care setting. 



Although group care is often 
envisioned as a place with 
consistent structure, youth in 
group care report they are 
far less likely to receive 
consequences for problem 
behavior than youth in high- 
quality treatment foster care. 



Chamberlain and her colleagues 
(1996) have shown that youth who are 
randomly assigned to MTFC and 
remain there for at least six months 
have better behavioral outcomes than 
youth placed in group care. Youth in 
group care often have deteriorating 
performance because of consistent 
exposure to deviant peers (Eddy & 
Chamberlain, 2000). This is consistent 
with basic research showing the acceler- 
ation of negative behavior that follows 
the convening of adolescent groups 
(Dishion, McCord & Poulin, 1999). 

Further, although group care is 
often envisioned as a place with consis- 
tent structure, youth in group care 
report they are far less likely to receive 
consequences for problem behavior 
than youth in high-quality treatment 



foster care (Chamberlain, Ray, & 
Moore, 1996). 

Such a brief review cannot do jus- 
tice to the many types of residential 
care or to the range of comparisons 
of residential care to other services. 
Family-focused approaches to residential 
care are emerging as clearly superior to 
others (Whittaker, 2000). Residential 
care that involves families appears to 
achieve better outcomes and, possibly, 
higher levels of cost-effectiveness 
(Hooper, Murphy, Devaney, & 
Hultman, 2000; Landsman, Groza, 
Tyler, & Malone, 2001; Romanksy, 
Lyons, Lehner, & West, 2003). These 
findings are consistent with social sci- 
ence research: “The cost-effectiveness 
of group interventions is retained if 
focus is on the parents and aggregating 
young adolescents is avoided” (Dishion, 
McCord, & Poulin, 1999; p. 762). 

The high level of staff turnover and 
requisite retraining also raises concerns 
about the cost-effectiveness of group 
care. Combined with higher costs for 
insurance and 24-hour care, as well as 
higher salaries for greater levels of pro- 
fessional staff in many facilities, the 
overall cost of residential care is two 
to five times higher than typical treat- 
ment foster care (Barth, 2002; MOLA, 
1999). If these resources were devoted 
to specialized treatment of children’s 
needs, and used to address family 
problems that detract from providing 
successful parenting, they might be 
more cost-effective. 

Out-of-home care resources are 
not well matched to children’s needs. 
Although there may be a difference 
between the level of problems facing 
children in group care and those in 
treatment foster care, among children 
in group care, there appears to be very 
little correspondence between the level 
of behavior problems of youth and the 
level of placement setting (Berrick, 
Courtney, & Barth, 1993). 



Nor does there appear to be much 
difference in programming or services 
between high-cost and lower-cost facili- 
ties (Coen, Libby, Price, & Silverman, 
2003). Indeed, many group care place- 
ments are made without assessing the 
placement’s ability to meet the needs of 
children (MOLA, 1999). 

Purchasing good care for children 
can be costly — strategies that adequate- 
ly address a range of child and family 
concerns of children in foster care and 
community settings are not readily 
available nor inexpensive. But simply 
because group care is more available 
than most emerging evidence-based 
alternatives is no justification for 
its use. Evidence in favor of group 
care’s effectiveness is quite weak, and 
it must be balanced against the real 
possibility that conventional group 
care contributes to a worsening of 
children’s conditions. 



Simply because group 
care is more available than 
most emerging evidence- 
based alternatives is no 
justification for its use. 



This conclusion must be under- 
stood with the knowledge that few 
cost-effective interventions are available 
for troubled children, and the evidence 
is inadequate to precisely order cost- 
effective interventions. Even the most 
promising interventions have not been 
widely tested by independent assessors 
in a range of settings. And not all stud- 
ies adequately account for children or 
families who leave the research before 
completion. 

Some children cannot or will not 
be served in treatment foster care or in 
family-based care because it’s not their 





parent’s preference or there are no 
locally available providers. In that case, 
group care may be the safest available 
alternative. 

Group care, with a family-focus, 
has the potential to be a valuable com- 
ponent of the children’s services arma- 
ment (Gibson & Noble, 1991). As cur- 
rently delivered, though, group care’s 
value for children and families is not 
justified by the cost to governmental 
agencies. But if group care were 
redesigned to build on the strengths of 
families, there’s a good chance that 
might change. 

Richard P. Barth, Ph.D., is Frank A. Danieis 
Professor and Chair of the Doctorai 
Program, School of Social Work, University 
of North Carolina, Chapel Hiil, NC. 
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In the /lexf Residential Group Care Quarterly Point/Counterpoint... 

Question: Should all organizations that provide training for behavior support and 
crisis management be required to certify their training model? 

Point: 

Certification is necessary and long overdue. Numerous training organizations pro- 
vide national and international behavior support and crisis management training, 
yet their individual training models are not accredited or certified. States have 
approached this dilemma in several different ways, from limiting the approved 
training models for use in their state to requiring all training models to be 
approved in advance. We’re left with a very disjointed system, and in some cases 
no system at all, to ensure these models are safe and effective. 

Counterpoint: 

Certification may not be the answer to ensure behavior support and crisis manage- 
ment training models are effective. More and more agencies are learning that 
behavior support and crisis management training in itself is not the answer 
to reducing the use of restraint and seclusion. Agencies have become much more 
creative in their approaches to reducing restrictive procedures. But by certifying 
these training models, we may create an atmosphere that views training as an 
end all. Certification may place more attention on the certification process and 
trainings, and less on what agencies are doing internally to reduce their use of 
restraint and seclusion. 
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Shame and Anger: 

Recognizing the Link Between 
Maltreatment and Malcontent 



By Marge Sullivan 



F or some years now, the 

Institute for the Study of Child 
Development/Pediatriacs has 
concentrated its research on the ways 
in which shame explains the diverse 
behavioral problems seen in children 
who have histories of maltreatment. 

Everyone agrees physical abuse 
and neglect are bad for children. But 
the sheer variety of problems reported 
for these children pose a significant 
challenge to those who serve them, 
plan the treatment programs, and 
direct social policy. Studies suggest 
that maltreated children can be aggres- 
sive, depressed, antisocial, or clingy. 
They may be withdrawn or sullen and 
bullying. 

It is critical to understand how 
such diversity arises in maltreated chil- 
dren, because if we understand the 
processes by which maltreatment leads 
to particular outcomes, we will do a 
better job of managing and treating 
behavioral problems. We will also be 
better able to identify those children at 
greatest risk and those who are more 
resilient. 

Models of Maltreatment 

Until recently, most research on the 
aftermath of maltreatment tacitly 
endorsed a trauma model of maltreat- 
ment. Typically, physical abuse or neg- 
lect were viewed as traumatic events 
and, therefore, the direct cause of sub- 
sequent behavioral difficulties. The 
form, amount, and severity of mal- 
treatment — or some combination — is 
still thought to explain the diverse 
problems seen in children. The trauma 
model is simple to understand, but it 



explains little and offers few practical 
strategies for treatment apart from 
prevention of the trauma in the 
first place. 

We believe a psychological model 
of maltreatment provides a more com- 
prehensive, process-oriented account 
of how maltreatment leads to poor 
behavior and suggests a number of 
treatment approaches. In the model we 
have articulated and are now evaluat- 
ing, the self-evaluative emotions by 
which children respond to their experi- 
ences of maltreatment and other person- 
al frustrations are the most important. 

These emotions, especially shame, 
may serve as the critical psychological 
process linking maltreatment to poor 
behavioral outcomes. Theory and our 
research suggest that shame and the 
self-blaming beliefs that accompany 
this emotion are critical mediators 
between a history of abuse and neglect 
and behavioral difficulties. 

Shame and Coping 

Shame is an emotion that emerges as 
soon as children begin to understand 
who they are. In their second year of 
life, most children develop an impor- 
tant mental capacity that allows them 
to recognize themselves in mirrors and 
other representations, to refer to them- 
selves, and to be self-aware. Lewis 
(1992) calls this capacity the “idea of 
me” or explicit self-consciousness. Its 
emergence makes possible a set of 
emotions important to human social 
life, such as embarrassment, jealousy, 
and empathy. By the time they are 3 
years old, children have begun to 
master an initial set of standards. 



rules, and goals as part of their early 
socialization. Additional mental abili- 
ties acquired during this period allow 
children to begin to compare their 
behaviors to these standards, rules, 
and goals, which may lead to shame. 
This emotion can be seen in a child’s 
body, face, and verbal behavior from 
the age of 3, and perhaps earlier. 

By age 4, children already exhibit 
individual differences in the amount of 
negative emotions shown in response 
to failure. Thus, even young children 
experience shame, and some develop 
a bias toward it, which undermines 
their adjustment. 

Shame is an intense, painfully 
negative emotion. When children 
experience shame, the most common 
response is to hide, disappear, or 
run away. All people try to cope with 
shame to avoid its pain, and many do 
so in maladaptive ways. When shame 
is not coped with adaptively, sadness 
and depression may result. Some indi- 
viduals try to rid themselves of shame 
through hostility and anger. They 
may become enraged and literally 
attack those whom they believe have 
shamed them. 

Shame’s Role in Sexual Abuse 

Shame plays an important role in 
adjustment following sexual abuse. 
When children are abused, their expe- 
rience of shame following discovery is 
what primarily determines the behav- 
ioral symptoms that follow, both 
immediately after the event and in the 
longer term. Children who showed 
more shame at the time of abuse 
discovery showed more depression. 
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eroticism, and effects of post-traumatic 
stress disorder one year later (Feiring, 
Taska & Lewis, 1999). 

Moreover, changes in shame and 
self-blame over time predict recovery fol- 
lowing the abuse. Children whose shame 
decreased were doing better a year after 
discovery, as measured by fewer behavior 
problems, less depression, and better self- 
esteem. Children’s adjustment one year 
after the abuse was not predicted by the 
severity of the initial abuse, but was a 
factor in their shame. (Feiring, Taska & 
Lewis, 2002). 

Shame in Physical Abuse 
and Neglect 

Shame may also explain some of the 
similarities in behavioral difficulties 
that commonly arise from physical abuse 
and neglect. The literature suggests that 
although depression may occur in the 
aftermath of both physical abuse and 
neglect, aggression and hostility appear 
to be more prevalent in abused children, 
especially boys, presumably because of 
their greater exposure to harsh punish- 
ment. Shame, however, may underlie 
both these difficulties. Children’s shame 
may interact with the form of maltreat- 
ment and produce either greater depres- 
sion or greater anger. 

Our work shows that maltreated 
children differ in their self-evaluative 
emotions because of the harsh treatment 
they receive. Preschoolers with histories 
of physical, sexual, and emotional abuse 
or neglect showed different emotional 
responses to success and failure relative 
to a matched control group (Alessandri 
& Lewis, 1996). The occurrence of phys- 
ical and excessively harsh, critical verbal 
punishment may cause children to feel 
shamed, unloved, and worthless. High 
demands and frequent, severe punish- 
ment are thought to be characteristic of 
physically abusive parents but may also 
occur among neglectful parents as well. 

To test whether shame influences the 
effect of maltreatment on children’s 
expressions of anger, we studied 86 chil- 
dren with substantiated case records of 
abuse, neglect, or both, as well as a 



matched community sample of 89 chil- 
dren without such histories. We counted 
all recorded allegations of physical abuse 
and neglect meeting standard definitions 
of maltreatment, whether they were sub- 
stantiated or not. Record abstractors 
were able to achieve 85% agreement. 
Disagreements were resolved by majority 
opinion following team review and query 
with child protective services. 

To quantify the amount of physical 
abuse, children were assigned a physical 
abuse rank as follows: controls were 
assigned 0, representing no known physi- 
cal abuse. Children with abuse records 
were scored as 1 plus the number of 
physical abuse allegations to arrive at 
their final score. Neglected children were 
assigned numbers using a similar system. 
Children’s shame, anger, and sadness 
in response to a mild failure experience 
(not completing several tasks by the 
time a bell sounded) were scored from 
videotape. 

Regression analyses examined the 
associations between level of physical 
abuse, shame, and anger. We also tested 
the shame- anger model for neglect and 
found that level of physical abuse was 
related to children's shame, and the 
amount of shame was related to their 
anger. Interestingly, maltreatment itself 
did not lead directly to anger. It led 
to shame, and shame lead to anger. 
Physically abused children, therefore, 
may be prone to anger because they 
feel shamed. 

When we examined the relation 
between levels of neglect, shame, and 
anger, only shame was related to chil- 
dren’s anger. Neglect was unrelated to 
shame. Shame thus appears to be a risk 
factor for increased anger in all maltreat- 
ed children. Physical abuse promotes 
shame, whereas neglect does not seem 
to do so, at least not directly. Shame 
in neglected children may be related to 
either aspects of child temperament, 
particular features of their parenting 
experience that are not a general feature 
of neglect, or some combination of these. 

High rates of harsh physical and 
verbal punishment seem to give children 



the message that they are very bad and 
unable to change. This punishment need 
not reach a criminal level of abuse to 
have such an impact on the child’s devel- 
oping self-system. Because badness is a 
core judgment, harsh punishment may 
signal profound failure of the self to the 
young child, thereby leading to shame. 
Our findings suggest that feelings of 
shame promote tendencies toward anger 
in maltreated children’s responses to 
even mild failures. These negative emo- 
tions are likely to underlie the hostility 
and aggression often described in mal- 
treated children. 

Our findings suggest that neglect and 
physical abuse show somewhat different 
patterns. Although shame seems to lead 
to greater feelings of anger in both forms 
of maltreatment, the physically abused 
child is at especially high risk, since 
abuse leads directly to shame, leading in 
turn to greater anger. 

In neglected children, anger may 
occur only in those who are shamed 
either because of their temperaments 
or particular life experiences that foster 
this emotion. This differential pattern 
by maltreatment suggests different inter- 
vention approaches may be needed to 
help children manage their emotions in 
the aftermath of physical abuse. 

Margaret W. Sullivan, PhD, is a professor 
at the Institute for the Study of Child 
Development, University of Medicine and 
Dentistry of New Jersey, Robert Wood 
Johnson Medical School, New Brunswick, NJ. 
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Legal Proceedings: 

NACC’s Children’s Law Conference in Vegas September ’04 



T he National Association of 

Counsel for Children (NACC) 
will hold its 27th National 
Children’s Law Conference, 
“Representing Children, Families, and 
Agencies in Child Welfare, Juvenile 
Justice, Custody, and Adoption 
Proceedings” in Las Vegas, Nevada, 
September 7-10, 2004. Tailored to 
those working in the fields of law, 
medicine, social work, and education, 
the conference generally draws more 
than 500 professionals working in 
child welfare, juvenile justice, and 
family law. 



Shay Bilchik, President & CEO of 
the Child Welfare League of America, 
will deliver the keynote presentation. 
Edward Lazarus, former editor of the 
Yale Law Journal and former clerk to 
Supreme Court Associate Justice Harry 
Blackman, will discuss the history and 
current status of the juvenile death 
penalty. John E.B. Meyers will address 
Crawford v. Washington and other 
developments in the law of evidence 
affecting children. Four different 
tracks offer sessions addressing Abuse 
& Neglect; Juvenile Justice; Custody, 
Visitation & Adoption; and Policy 



Advocacy. A pre-conference session on 
developing children’s law office prac- 
tices and procedures will be held on 
September 7th. 

The conference is supported by the 
American Bar Association, the 
National Council of Juvenile and 
Family Court Judges National CASA, 
CWLA, and many others. Continuing 
education credits are available. For 
more information including a full 
schedule of conference events, contact 
NACC at 888/828-NACC or visit 
www.NACCchildlaw.org. 




AMERICA 



440 First Street, NW, Third Floo 
202/S38-2952 ■ Fax 202/638- 



DC 20001-2035 



CHILD WELFARE LEAGUE OF AMERICA 
Mid-Atlantic Region Training Conference 

^Rising m the 

Challenge 

Strengthening Community Partnerships for Healthy Children, Families, and Staff 



The Mid-Atlantic Training Conference in 
Philadelphia, entitled Rising to the 
Challenge: Strengthening Community 

Partnerships for Healthy Children, Families, 
and Staff, will highlight the strength of 
working together in partnership as individ- 
uals and families, and helping, inviting, 
and involving representatives from a broad 
spectrum of the service community. 

From September 20—22, workshop 
sessions will focus on collaboration as a 
value and an important aspect of best 
practice in serving children and families; 
provide training and networking opportu- 
nities; provide time for the exchange of 
ideas between parents, young people, and 
professionals representing various disci- 
plines; and highlight how child welfare 
effectively operates as an integral part of 
communities. For more information, visit 
www.cwla/conferences. 



September 20-22, 2004 

Hyatt Regency Philadelphia at Penn's Landing 
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American Association or 
Children's Residentfal Centers 
Uresents its 4Sth Annual Meeting 
October 13-16, 20D4 
R&voiutiontzlttg 
Mapping Out New Triais 
Boston, Massachusetts 



Featured speakers mc/ude: 
Lirda Sanford, IMSW, LFCSW, 
Charlie Applestein, MSW 
and Mary Caesar-Hurphy, PhD 




CALL lUP «.XH!PII4IPP 



ALLINTHEFAMILV 

Achieving Excellence in Adlaplion 
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iln rl;stu;.a[il [|OTiiM i| ji ii xinTliii ikJ 



Gik-AEBBi; li^ ki 

j-Tri Hi ii wn i 

bling bitt ILilfiiiHi iii ' 

R«i nsBTiitolcinLtti 6w ifjsrd. [ 



iBOsfp^ 13’^^ 
'‘fefWt 



Far inliimjliHi oi Bh'Jitisifig rr Eidiitiriig EMTiNkfiffBiif'IsiiflirBl BdwrtHii^££wla.ir|. 
Far iriiimaUm rmialnlliiiilni hr Iht r^uiiiimiA B-fi«l:adg0nf<3^Hla.aii]. 



16 





